Skip to content

fix(schedule): bound AI task runs and wait for SOP outcomes - #1080

Merged
Robert0622 merged 9 commits into
mainfrom
codex/schedule-timeout-sop
Nov 21, 2026
Merged

Robert0622 merged 9 commits into
mainfrom
codex/schedule-timeout-sop

Conversation

@Robert0622

@Robert0622 Robert0622 commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

The v0.4.0 scheduled assistant had no per-execution deadline. SOP dispatch returned immediately, so unattended prompts that needed an outcome could keep querying run details and logs indefinitely.

Changes

  • Add an execution timeout for AI schedules and project subscriptions. Existing jobs default to 30 minutes and can be set to 1–240 minutes. Expiry stops the AI turn and its pending tool calls and records a failed execution with a timeout reason. Dispatched SOPs have their own lifecycle.
  • Add the read-only runs_wait tool. It waits on the exact run ID using a lightweight status query, returns when the run completes or needs human action, and yields after at most 25 seconds per call. Scheduled prompts use it when they need to wait for an SOP outcome.

Validation

  • Ran the original TestScheduledAssistantHasNoDeadline at v0.4.0 to confirm the missing execution deadline.
  • Passed the Go regressions for schedule configuration, timeout cancellation, SOP state and access checks, and the OpenAPI/MCP contract.
  • Passed the scheduled task web tests and all 401 chatbot tests. The chatbot build, go vet, public content check, and Gitleaks scan also passed.
  • Passed the Go suite across non-browser tests and the five separate Chromium tests. The scheduled gateway regression checks tool availability beyond the former five-minute TTL and within the execution deadline.
  • Ran the public Playwright suite against an isolated MySQL installation. 23 passed and 6 were skipped because they require first-run setup or real model credentials.

Live CodeBuddy regression

Real DeepSeek V4.1 Flash was used for the timer prompt and the SOP Agent node on isolated MySQL instances. The Agent wrote a marker artifact, which the following script verified.

Scenario Observed result
v0.4.0, slow SOP, requested one-minute timeout The saved job omitted the timeout. The timer prompt was still running at 65 seconds and used 37 runs_get calls before reporting SOP success at 121.7 seconds.
Initial fix (7382983f), slow SOP, one-minute timeout The timer prompt stopped at 60.1 seconds and recorded an execution timeout. It issued 2 runs_wait calls, including the wait that was cancelled at the deadline. The dispatched SOP later succeeded.
Initial fix (7382983f), short SOP The timer prompt used one runs_wait call and reported the correct success state in 36.8 seconds.

The v0.4.0 run eventually reached the correct outcome in this fixture. The comparison reproduced the omitted deadline and repeated reads.

Follow-up live regression

A fresh timer run exposed another path on 7382983f. The model submitted runs_start as a final reply draft. The schedule recorded success after 15.0 seconds, applied the draft after the CLI exited, and never checked the SOP outcome.

Commit 0c3e528b keeps scheduled SOP dispatch in the active turn. The gateway directs the model to call the dispatch tool, and text recovery rejects a leftover dispatch draft before it can run. The added regression tests cover start, issue dispatch, retry, and the recovered-draft path.

We repeated both timer cases with CodeBuddy on DeepSeek V4.1 Flash and isolated MySQL. Each dispatched one SOP. A real Agent wrote a unique marker in each run, a script checked it, and both the marker and completion artifact were read back from storage.

Timer case on the updated fix Observed result
Short SOP One runs_wait returned succeeded. The timer reported the correct run ID and finished in 40.398 seconds.
Slow SOP, one-minute timeout The test prompted the model to try the terminal draft first. The gateway rejected it, then received one direct start and two runs_wait calls. The deadline cancelled the second wait (HTTP 408) and the timer recorded EXECUTION_TIMEOUT at 60.101 seconds. The independent SOP later succeeded.

Neither timer called runs_get. A later read found the same persisted generation sequence and update timestamp for both completed timer executions.

@Robert0622

Robert0622 commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator Author

定时任务超时与 SOP 终态实测

复测通过。 在提交 0c3e528b 上,由调度器自动触发两条提示词任务。定时助手和 SOP 的 Agent 节点都使用 CodeBuddy 的 DeepSeek V4.1 Flash。每个 SOP 都由模型写入一个唯一标记,再由脚本校验。两条运行均只派发一次,已从 MySQL 读回并核对标记和完成产物。

版本与场景 实际结果
v0.4.0,设置 1 分钟超时 保存后丢失超时设置。任务对同一运行调用 runs_get 37 次,在 121.7 秒后才报告 SOP 成功。
初版修复 7382983f,模型把启动 SOP 放在最终回复草稿里 定时任务在 15.0 秒就记为成功,此时 SOP 仍在执行,会话已经结束。
当前提交,短 SOP 一次 runs_wait 收到 succeeded,38.046 秒后定时任务报告正确的运行 ID 和终态。
当前提交,长 SOP,超时设为 1 分钟 网关拒绝最终回复里的启动草稿,模型随后直接启动一次 SOP。第二次 runs_wait 在截止时被取消(HTTP 408)。定时任务在 60.156 秒记录 EXECUTION_TIMEOUT,已派发的 SOP 随后独立成功。

以上两条当前提交的运行均未调用 runs_get。前一轮独立复测也观察到短任务在 40.398 秒正确结束、长任务在 60.101 秒超时,超时后再次读取该会话,事件序号和更新时间保持稳定。此次验证覆盖了等待和超时行为。线上磁盘写满的具体原因还需事发时的数据库记录确认。

本轮执行记录(两条均为定时触发)

定时任务执行记录,长任务按时超时,短任务报告正确的 SOP 运行 ID 和成功状态

已保存的 1 分钟超时和 Flash 模型配置

长任务编辑页中的 CodeBuddy、DeepSeek V4.1 Flash 和 1 分钟超时

草稿回退、单次启动,以及超时取消等待的网关记录

网关拒绝结束会话的草稿,随后记录一次启动和两次等待,最后一次返回 HTTP 408

定时会话超时后,独立 SOP 的最终结果

长 SOP 最终成功,Flash 写入和脚本校验两个节点均已完成

@Robert0622 Robert0622 left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0c3e528b 的等待和执行超时通过独立复测。用 CodeBuddy iOA 的 DeepSeek V4.1 Flash,通过定时助手读取隔离测试库中预置的 SOP 状态,并在 Go HTTP 路由统计实际请求。

  • SOP 为 waiting 时,定时助手调用 runs_wait 1 次,18.439 秒后报回匹配的运行 ID 和 waiting。
  • SOP 保持 running、定时任务时限为 1 分钟时,助手发起 2 次等待,第 2 次由截止时间取消。60.061 秒后该轮生成记录为 failed / EXECUTION_TIMEOUT,SOP 状态仍为 running。

定时任务配置、旧任务默认时限、超过 5 分钟的网关会话、取消等待及终态草稿回归也已通过。当前合并阻塞在 internal/httpapi/assistant_schedule.go 和 internal/httpapi/assistant_tasks.go。与 main@6a331fb5 解决冲突时,需要同时保留本 PR 的 Timeout 和主线的 ReportUsage 回调及其执行上下文,否则定时任务会丢失超时或用量统计。

@Robert0622 Robert0622 left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

复审通过 · 4a94f4d7

用真实 iOA CodeBuddy(deepseek-v4.1-flash)通过定时助手访问隔离测试库。SOP 为 waiting 时,助手发出 1 次 runs_wait,38.638s 后返回匹配的运行 ID 和状态。SOP 持续 running、任务时限为 1 分钟时,助手发出 2 次等待,1 次由截止时间取消,1m0.201s 后生成记录为 failed / EXECUTION_TIMEOUT。两次均记录实际模型和用量,SOP 自身状态与测试前一致。超时配置、取消和工具会话回归也已通过。

已核对本提交的差异和 GitHub 检查,当前没有发现阻塞合并的问题。

@crispinhu27 crispinhu27 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

初审结论:REQUEST_CHANGES(head 4a94f4d7,base main@5fadefca)

审查范围:完整 diff,包括执行超时(executeAssistantTask / runScheduleJobPromptResult / generation 超时分类)、runs_wait 端点与工具网关免锁、定时会话 SOP 派发草稿拦截、schedule 配置校验、Web 表单、OpenAPI/MCP 契约,以及相关调用方和权限链路。

已执行的验证(隔离环境,未接触任何凭据):

  • go build ./... 和 go vet(touched 包)通过;internal/domain、internal/repo、web 测试通过。
  • internal/httpapi 全量测试:只有 TestMainBotBrowserStartsAndDrivesRunBrowser 和 TestBrowserStreamPushesFramesAndRunsCommands 失败。在 merge-base 5fadefca 上同样失败,是本容器 Chromium 的环境问题,与本 PR 无关;GitHub 上的 browser 检查是通过的。
  • webtest sync-schedule.test.mjs 17/17 通过。
  • 用本 head 构建的服务连隔离的 MariaDB 实际运行,复现了下面行内评论里的问题,并做了带 projectId 的对照。

阻塞问题:1 个 [P2],见行内评论(runs_wait 对不带 projectId 启动的 SOP 返回 403)。

未验证:没有用真实模型做端到端的定时提示词回归(需要模型凭据);超时取消路径只依据单测和 PR 描述中的实测结果,本次没有另行复测。

Comment thread internal/httpapi/run_wait.go Resolved

@crispinhu27 crispinhu27 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

复审结论:APPROVE(head 8b77d152,base main@edc27c0,PR 内已合入 main@fe053a1)

历史问题

  • [P2] runs_wait 对不带 projectId 启动的 SOP 返回 403:FIXED。在隔离 MariaDB 上用真实服务复测,原触发条件下上一版 head 4a94f4d7 返回 403,本 head 返回 200,详见原线程。

本次审查范围

  • 612e092d:waitRun 的项目解析顺序是 executionContext → issue 项目 → 冻结的 workflow snapshot(RunWorkflowProjectID),以及新增的回归测试。
  • 合并提交 8b77d152:--remerge-diff 显示只有 internal/httpapi/agent_tool_mcp.go 一处冲突。解决后同时保留了本 PR 的 validateScheduledRunDrafts 和主线 #991 的 vetSOPDesignerReply。合并前后 PR 的净 diff 一致(25 个文件,+529/−35)。

已执行的验证(隔离环境,不接触任何凭据)

  • go build ./...、go vet(httpapi/repo/domain)通过。internal/domain、internal/repo、internal/schedule、web 测试通过。
  • internal/httpapi 全量测试只有 TestMainBotBrowserStartsAndDrivesRunBrowser 和 TestBrowserStreamPushesFramesAndRunsCommands 失败。这两个用例在 merge-base 5fadefca 上同样失败,原因是本容器的 Chromium 环境,GitHub 的 browser 检查是通过的。TestScheduledCodexCacheWriteUsage 偶发失败,merge-base 上 10 次失败 6 次,本 head 10 次失败 3 次,属于已有的不稳定用例。
  • 执行超时运行验证:用一个接受 turn 后不再结束的假 codex CLI(AGENTFLOW_CODEX_BIN,合成凭据)创建 timeoutMinutes=1 的定时任务并立即运行。09:16:52 开始,09:17:52 终止。任务状态为 failed / 定时任务执行超时,generation 记录为 failed,failure.code=EXECUTION_TIMEOUT,假 CLI 进程已被回收。
  • GitHub 检查:本 head 的 platform-required、e2e-required 等 16 项检查全部通过。

未覆盖:没有用真实模型跑端到端的定时提示词回归,因为需要模型凭据。超时和 runs_wait 的服务端行为已经按上面的方式实际运行验证。

@Robert0622
Robert0622 merged commit 42541ef into main Nov 21, 2026
16 checks passed
@Robert0622
Robert0622 deleted the codex/schedule-timeout-sop branch November 21, 2026 09:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants